Image processing apparatus

ABSTRACT

There is provided an image processing apparatus that can implement image recognition processing on all of objects to be recognized, and can reduce a load of capturing and transferring images. The image processing apparatus includes: an image capturing unit that captures image information picked up by an imaging element; a processing region setting unit that sets a plurality of processing regions for the image information; a processing sequence/frequency determination unit that determines at least any one of a sequence, a frequency, and a timing of capturing the respective image information in a plurality of set processing regions, and at least any one of a sequence, a frequency, and a timing of processing the respective image information; and an image processing unit that captures the image information for each of the processing regions according to the sequence, the frequency, or the timing which has been determined, and processes the captured image information according to the sequence, the frequency, or the timing which has been determined.

This application is the National Phase of PCT/JP2012/059184, filed Apr.4, 2012, which claims priority to Japanese Application No. 2011-089702,filed Apr. 14, 2011, the disclosures or which are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The present invention relates to an image processing apparatus thatcaptures, transfers, and processes an image.

BACKGROUND ART

In recent years, in-vehicle cameras intended for drive assistance, andsecurity cameras intended for assistance of crime prevention or criminalinvestigation have been increasingly developed and widespread. Thosecameras not only allow a user to view picked-up images, but also areequipped with a function of automatically detecting lanes, vehicles,persons, and the motion of the persons through image processing. Theimage processing needs to capture data of a large number of pixels whichis generally hundreds of thousands of pixels or more from an imagingelement, and transfer and process the data. Further, in recent years, ahigh resolution of the imaging element is advanced, and a load ofcapturing, transferring, and processing the image has a tendency to befurther increased. Also, in a stereo camera that measures threedimensions with the use of two cameras, a system in which a plurality ofcameras is installed anterior to, posterior to, and lateral to a vehicleto monitor a periphery of the vehicle, or a security camera system inwhich a plurality of cameras is installed at different positions,because there is a need to process a plurality of images, the load ofcapturing, transferring, and processing the image is further increased.

Under the circumstances, a technique for limiting a region in which theimages are captured, transferred, and processed is required. As thetechnique for limiting the region in which the image is processed, amethod disclosed in Patent Literature 1 has been proposed. This is atechnique in which if no vehicle is present at a short distance, acenter portion of the images is processed without being thinned out, andif a vehicle is present at the short distance, the images are thinnedout, and the overall images are processed. In the technique, one of theimage of the unthinned center portion, and the image of the thinnedoverall images is selected according to the presence or absence of thevehicle. Alternatively, any one of a right region of the image and aleft region of the image is selected on the basis of indicatorinformation.

-   [Patent Literature 1] JP-A-2000-251080

SUMMARY OF INVENTION Technical Problem

However, in the technique of Patent Literature 1, whether the images arethinned out, and the overall images are processed, or the center portionof the images are processed without being thinned out, is selected onthe basis of whether the vehicle is present, or not. For that reason,when a variety of objects to be recognized such as lanes, pedestrians,or vehicles are present, there arises such a problem that the image tobe used is different, and a part of image recognition processing cannotbe implemented. Also, because the overall picked-up images are capturedin a memory that stores data to be processed by a CPU therein, and onlya processing region is limited, there arises such a problem that a loadof capturing and transferring the image cannot be reduced.

An object of the present invention is to provide an image processingapparatus that can implement, if a plurality of objects to be recognizedis present, the image recognition processing on all of the objects to berecognized, and can reduce the load of capturing and transferring theimages.

Solution to Problem

In order to solve the above problem, according to the present invention,there is provided an image processing apparatus including: an imagecapturing unit that captures image information picked up by an imagingelement; a processing region setting unit that sets a plurality ofprocessing regions for the image information; a processingsequence/frequency determination unit that determines at least any oneof a sequence, a frequency, and a timing of capturing the respectiveimage information in the plurality of set processing regions, and asequence, a frequency, and a timing of processing the respective imageinformation; and an image processing unit that captures the imageinformation for each of the processing regions according to thesequence, the frequency, or the timing determined by the processingsequence/frequency determination unit, and processes the captured imageinformation according to the sequence, the frequency, or the timingdetermined by the processing sequence/frequency determination unit.

Advantageous Effects of Invention

There can be provided the image processing apparatus that can implement,if a plurality of objects to be recognized is present, the imagerecognition processing on all of the objects to be recognized, and canreduce the load of capturing and transferring the images.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an image processing apparatusaccording to a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating the image processing apparatusaccording to the first embodiment of the present invention.

FIG. 3 is a block diagram illustrating an image processing apparatusaccording to another embodiment of the present invention.

FIG. 4 is a diagram illustrating an example of setting a processingregion and capturing the processing region in the image processingapparatus according to the present invention.

FIG. 5 is a diagram illustrating an example of conducting pipelineprocessing on capturing of the processing region and processing of thecaptured processing region in the image processing apparatus accordingto the present invention.

FIG. 6 is a diagram illustrating a processing flow of the imageprocessing apparatus according to the first embodiment of the presentinvention.

FIG. 7 is a diagram illustrating an example of a table for determining asequence, a frequency, and a timing of processing the respectiveprocessing regions in the image processing apparatus according to thepresent invention.

FIG. 8 is a block diagram illustrating an image processing apparatusaccording to a second embodiment of the present invention.

FIG. 9 is a block diagram illustrating the image processing apparatusaccording to the second embodiment of the present invention.

FIG. 10 is a block diagram illustrating a principle for measuring adistance by the image processing apparatus of the present invention withthe aid of two cameras.

FIG. 11 is a diagram illustrating an example of setting a processingregion and capturing the processing region in the image processingapparatus according to the second embodiment of the present invention.

FIG. 12 is a diagram illustrating a processing flow of the imageprocessing apparatus according to the second embodiment of the presentinvention.

FIG. 13 is a diagram illustrating one example of conducting pipelineprocessing on processing of the respective processing regions in theimage processing apparatus according to the second embodiment of thepresent invention.

FIG. 14 is a diagram illustrating an example of a correspondencerelationship between image processing functions and the respectiveprocessing regions in the image processing apparatus according to thepresent invention.

FIG. 15 is a diagram illustrating another example of conducting pipelineprocessing on processing of the respective processing regions in theimage processing apparatus according to the second embodiment of thepresent invention.

REFERENCE SIGN LIST

-   1 image processing apparatus-   2 camera unit-   4 camera-   5 image input means-   6 CPU-   7 ROM-   8, 9 memory-   20 image capturing unit-   21 processing sequence/frequency control unit-   22 processing region determination unit-   23 image processing unit-   24 image transfer unit

Description Of Embodiments

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

(First Embodiment)

FIG. 2 is a block diagram illustrating an example of a basicconfiguration of an image processing apparatus according to a firstembodiment of the present invention.

The first embodiment is directed to an image processing apparatus and amethod of controlling the image processing apparatus, which control atleast any one of a processing region, a frequency, a sequence, and atiming of images captured from a camera which is an imaging element, andat least any one of a processing region, a frequency, a sequence, and atiming of processing the captured image.

An image processing control according to the present invention isapplied to the image processing apparatus 1 which is an imaging deviceas illustrated in FIG. 2, and realizes a function of picking-up asurrounding environment by means of a camera 4 (imagining element)provided in the image processing apparatus 1, and processing thepicked-up image. In this situation, two or more cameras may be provided.Hereinafter, an exemplary configuration of FIG. 2 will be described.

The image processing apparatus 1 includes the camera 4 which is theimaging element, a CPU 6 which is processing means for processing imagespicked by the camera 4, a memory 9 which is recording means for use inthe CPU 6, a ROM 7 which is recording means of program and data, andimage input means for capturing the images together with controlling ofthe camera 4, which are connected to each other by a bus 11. The aboveconfiguration is not a necessary requirement, but may be replaced with aconfiguration in which an image processing LSI is further installed inaddition to the CPU 6 to process a part of processing by a dedicatedLSI, or a configuration in which not only one RAM, but a plurality ofRAMs such as a RAM for the image processing LSI is used.

Also, as illustrated in FIG. 3, the image processing apparatus 1 may beconfigured to be divided into a camera unit 2 having the camera 4 andthe image input means 5, and a processing unit 3 having the CPU 6, thememory 9, and the ROM 7. In this case, the camera unit 2 and theprocessing unit 3 may be equipped with an I/F 12 and an I/F 13 which cancommunicate data with each other, respectively, and the I/F 12 and theI/F 13 may be connected to each other through data transfer means 14.

The camera 4 of the image processing apparatus 1 is equipped with, forexample, a CCD or a CMOS imaging element. When it is assumed that thenumber of pixels in the imaging element is 2 million pixels, and data ofone pixel has three bytes of R(red), G(green), and B(blue) in total, apiece of image data has 6 megabytes. In this case, if image processingis executed 60 times per second, a data traffic of the bus 11 forstoring the picked-up image in the memory is 360 megabytes/sec. The bus11 needs to transfer the data even when the CPU 6 processes the datastored in the memory, in addition to capturing of the image, and atransfer load of the bus 11 becomes a problem. Also, because a built-indevice is lower in CPU performance than a PC, there arises such aproblem that if the image data stored in the memory 9 is large, theimage data cannot be processed within a given time by the CPU, inaddition to the bus transfer load.

The above data size for each image is different depending on the numberof pixels in the image, the data size per pixel, a format of the image,or the like, and the bus transfer load is different depending on a cyclein which the images are transferred, and processed. For that reason, theabove numerical values are exemplary.

In order to reduce the bus transfer load, there is a need to capture agiven region of the image, or an image into which an original image isreduced in size, instead of capturing of the overall image.

Also, the processing load of the CPU can be reduced by dividing theimage into several regions, and capturing those regions so as to processthe captured image while capturing a certain region, in addition tocapturing a given region of the image or the image obtained by reducingthe original image instead of capturing of the overall image.

In the image processing apparatus 1 according to the present invention,in order to realize the above configurations, control programs of theimaging device according to this embodiment are stored in the ROM 7, andwhen a power supply turns on, the CPU 6 executes those programs tocontrol the image processing apparatus 1, and to realize a sequence offunctions for processing the picked-up image.

FIG. 1 illustrates a detailed block diagram of the CPU 6 in the imageprocessing apparatus 1. The image input means 5 of FIG. 2 has an imagecapturing unit 20, and the CPU 6 has a processing sequence/frequencycontrol unit 21, a processing region determination unit 22, and an imageprocessing unit 23. A sequence of functions is realized by thoseprocessing.

The image capturing unit 20 has a function of capturing the image pickedup by the camera 4 which is the imaging element, and storing the imagein the memory 9 as well as, for example, a function of designating aregion for capturing the image in the imaging element of the camera 4,and a function of setting a shutter or gain of the imaging elementrelated to the exposure, and adjusting the timing. Also, the imagecapturing unit 20 may have a function of conducting an edge emphasis, anoise removal, and a correction of brightness of the image beforestoring the image captured from the camera in the memory 9. Also, incapturing the image, the image capturing unit 20 may divide the capturedimage into the reduced image, an unreduced image, or a partially cut-outimage, and store those images in the memory 9.

The processing sequence/frequency control unit 21 has a function ofcontrolling at least any one of the sequence, the frequency, or thetiming of capturing the images in the processing regions, and storingthe images in the memory 9 for a plurality of processing regions of theimage captured by the camera 4, and at least any one of the sequence,the frequency, or the timing of allowing the CPU 6 to process the imagessaved in the memory 9.

The processing region determination unit 22 has a function ofdetermining the region of the image used for processing on the basis ofan image processing result of the image processing unit 23 to bedescribed later, or a region that can be obtained according to thefunction of the image processing or a preset region.

The image processing unit 23 has a function of processing the imagecaptured in the memory 9 by the processing region determination unit 22according to the sequence, the frequency, or the timing determined bythe processing sequence/frequency control unit 21, for the processingregion determined by the processing region determination unit 22. Theimage processing includes processing of removing noise specific to theimaging element, or noise generated when the image is picked up under adark environment such as a night, processing of extracting only aportion in which a value of the pixel is changed through a knowntechnique such as a background differencing technique, or processing ofdetecting a person through known pattern identification processing.

With the use of those functions, for example, as illustrated in FIG. 4(a), in the vicinity of an exit of a tunnel, an image center is set as afirst processing region, and a region into which the overall image isreduced is set as a second processing region. As illustrated in FIG. 4(c), the first processing region is captured in the memory 9 at a time t,and thereafter the image into which the overall image is reduced can becaptured in the memory 9 at a time t+1. For example, when it is assumedthat a lateral size and a longitudinal size are set to ½ of the overallimage in the first processing region, and the overall image is reducedto ½ in the second processing region, a bus transfer load and aprocessing load of the CPU 6 can be halved as compared with a case inwhich the overall image is captured.

Also, as illustrated in FIG. 4( b), when a shutter is released to startexposure before the time t, a next shutter is released after the timet+1, and electric charge accumulated in the respective pixels istransferred to capture the image, the electric charge of the pixelsother than the first processing region continues to be accumulated. Forthat reason, a dark image can be captured in the first processingregion, and in the second processing region, the dark image can becaptured in the region where the image has been captured in the firstprocessing region, and a bright image can be captured in the otherportions. As a result, a phenomenon that the electric charge accumulatedin the pixels at the exit of the tunnel is saturated can be suppressed,and in a region where the lane or the like is caught, a bright image inwhich the electric charge is accumulated from an exposure start to thetime t+1 can be picked up. Also, instead of transferring the electriccharge accumulated in the respective pixels to capture the image, animaging element that can capture the images by reading a voltage of theelectric charge accumulated in the respective pixels is used. In thiscase, because the electric charge accumulated by the image capturingprocess is not reset, the overall second processing region can becaptured as an image picked up for the same exposure time. In this way,the respective processing regions are captured at different timingduring one imaging cycle, resulting in such an advantage that the imagesin a plurality of processing regions can be picked up for a plurality ofexposure times in one imaging time.

FIG. 5 is a diagram illustrating an example of conducting pipelineprocessing on capturing of the processing region and processing of thecaptured processing region in the image processing apparatus 1. FIG. 5illustrates an example in which lamp detection processing of a head lampor a tail lamp is conducted with the use of the first processing region,and lane (lane) recognition processing is conducted with the use of thesecond processing region.

In this example, the second processing region is captured while thefirst processing region is captured, and the image of the firstprocessing region is being processed by the image processing unit withthe result that a time required for capturing the image and processingthe image can be reduced. It is apparent that the image captured in thepast can be processed while the first processing region is beingcaptured, and three or more processing regions can be provided.

As described above, the image processing apparatus 1 configured asdescribed above can control, with the execution of the control programof the image processing apparatus by the CPU 6, at least any one of theregion of the captured image, the captured timing, the frequency, andthe sequence, and at least any one of the timing, the frequency, and thesequence of processing the captured image. As described above, a part orall of the image capturing unit 20, the processing sequence/frequencycontrol unit 21, the processing region determination unit 22, and theimage processing unit 23 may be executed by the processing unit 3.

In this example, in the image processing apparatus 1 and the controlmethod of the image processing apparatus as described above, a sequenceof processing flow for controlling the image processing apparatus willbe described with reference to a flowchart.

A sequence of processing illustrated in FIG. 6 starts when a powersupply turns on, and is repetitively conducted until the power supplyturns off. Alternatively, there may be applied a configuration in whichthe image processing is completed after the image processing has beenexecuted by a given number of time, or a configuration in which asequence of processing starts when a program is executed by an externaldevice not shown, and the program stops when stop processing isconducted by the external device.

First, when the power supply turns on, the control processing of theimage processing apparatus and the image processing program are executedby the image processing apparatus 1, and initialization processing atthe time of startup such as the initialization of the memory 9 isexecuted (Step 101). In an example of FIG. 6, the image processing unit23 conducts leading vehicle detection processing and lane detectionprocessing. Then, it is determined whether it is initialization timingduring execution of the program, or not (Step 102). If it is the timingfor executing the initialization in Step 103, a flow proceeds to Step111, and if it is not the timing for executing the initializationprocessing in Step 103, the processing in Step 104 is executed. In theinitialization processing, coordinates, image sizes, andreduction/enlargement ratios of the image in the first processing regionand the second processing region are initialized (Step 111).

In the timing determination of the initialization processing, forexample, the image processing apparatus 1 is attached to a mobile objectsuch as a vehicle, a road on which the vehicle is traveling is estimatedby a steering angle sensor, a wheel speed sensor, a vehicle speedsensor, a gyro sensor, a yaw rate sensor, a GPS sensor, or a mapdatabase not shown, and it is determined whether it is the timing of theinitialization processing, or not. For example, it is determined that asubject vehicle takes a left turn or a right turn on the basis ofinformation obtained from the steering angle sensor, and theinitialization processing is conducted when the subject vehicle takesthe left turn or the right turn. This is because when the subjectvehicle takes the right turn or the left turn, the subject vehicletravels on a road different from the road environment where the subjectvehicle has traveled, and therefore the initialization of the processingregion is effective. Also, the image processing apparatus may beinstalled other than the vehicle, for example, may be installed within abuilding or an elevator. A timing at which a person, a vehicle, or anobstacle within a viewing field of the camera falls outside the viewingfield, or a timing at which the person goes out of the elevator or thebuilding may be determined as the initialization timing.

Which area of the image picked up by the camera 4 is to be processed isdetermined by the processing region determination unit 22 (Step 104). Inthis processing, the processing region is determined with the use of thecoordinates of the starting point, the coordinates of the ending point,and the reduction ratio of the processing region, which have been savedin the ROM 7 or the program as constants in advance by theinitialization processing immediately after the image processingapparatus 1 has started, and thereafter the processing region can bedetermined with the use of the image processing result of the imageprocessing unit 23. For example, when the vehicle is detected byprevious image processing, the processing region can be determined onthe basis of a position and a size of the vehicle on the image, whichhave been previously detected. Processing for determining the processingregion on the image with the use of the image processing resultcorresponds to Step 110.

Thereafter, at least any one of the sequence, the frequency, or thetiming of processing the respective processing regions of the imagepicked up by the camera 4 is determined by the processingsequence/frequency control unit 21 (Step 105). The sequence, thefrequency, or the timing can be determined according to the exposuretime, the coordinates of the starting point/ending point, the capturingtime, the processing time, and the reduction/enlargement ratio of therespective processing regions, for example, as illustrated in FIG. 7.

For example, the processing region short in the exposure time iscaptured at the time t, and the processing region long in the exposuretime is captured at the time t+1. As described above, during oneexposure period, the first processing region is first captured tocapture the dark image, and thereafter the second processing region canbe captured as the bright image as illustrated in FIG. 4. That is,plural pieces of image information of the processing regions differentin the exposure can be captured during one exposure period. When it isassumed that the exposure time of the first processing region is T1, andthe exposure time of the second processing region is T2, if T1<T2 issatisfied, the sequence and the timing are determined so that the firstprocessing region is captured at timing when T1 has been elapsed fromthe exposure start, and thereafter the second processing region iscaptured at timing when T2 has been elapsed from the exposure start.Also, as a result of processing the first processing region in the past,if no vehicle is detected, the first processing region is captured, theprocessing frequency is reduced, and the first processing region may beso set as not be captured in the present exposure.

Subsequently, the first processing region determined in Step 104 iscaptured according to the sequence, the frequency, or the timingdetermined in Step 105 (Step 106). For example, as illustrated in FIG.4( a), the first processing region is set in the image center, andcaptured at a timing when T1 has been elapsed from the exposure start asdescribed above. Alternatively, as illustrated in FIG. 4( b), the firstprocessing region is set in an upper portion of the image, and aluminous body such as a traffic light can be captured as an image shortin the exposure time.

Then, the captured image is processed to detect a leading vehicle (Step107). In this processing, the leading vehicle is detected, for example,with the use of a pattern identification based on the brightness valueof the image, and an optical flow that obtains a movement vector of therespective pixels from the past image and the present image.

Thereafter, as in Step 106, the second region is captured in the memory9 in Step 108, and the captured image is processed to detect the lane(Step 109). The lane can be detected by a known technique.

Finally, the first processing region and the second processing region tobe processed in a subsequent picked-up image are updated on the basis ofthe image processing result (Step 110). This can be determined, forexample, according to the coordinates and the size of the detectedvehicle on the image if the vehicle is detected in Step 107.Specifically, when it is assumed that the coordinates of a viewpoint ofthe region of the vehicle are Cx and Cy, and the size is Cw and Ch, theviewpoint of the first processing region is set to Cx−α1, Cy−α2, Cw+α3,and Ch+α4 so as to include the region of the vehicle. α1 to α4 aremargins in the periphery of the region of the vehicle, and may be set asconstants, or may be calculated on the basis of a travel distanceobtained by tracking the leading vehicle in a time direction, or themotion of the subject vehicle obtained with the use of the steeringangle sensor, the vehicle speed sensor, the gyro sensor, or the yaw ratesensor.

The method described above uses one camera. However, it is apparent thatthis embodiment can be applied to even a case in which two or morecameras are present. When two or more cameras are present, because atotal of the data size of the image becomes larger than that when onecamera is used, the advantageous effect of the present invention islarge. Also, when two or more cameras are present, the processing regiondetermination unit may set the processing region for the images pickedup by a plurality of cameras.

(Second Embodiment)

Subsequently, the image processing apparatus 1 according to a secondembodiment of the present invention will be described in detail withreference to the drawings.

In the configurations of the image processing apparatus, and a controlmethod of the image processing apparatus according to the secondembodiment, the same configurations as those of the above-mentionedfirst embodiment are denoted by identical symbols in the figures, and arepetitive description will be omitted.

An example of a basic configuration of the image processing apparatus 1according to the second embodiment is illustrated in FIG. 9. The detailswill be described later.

The second embodiment is directed to the image processing apparatus 1that captures images from a camera 4 a and a camera 4 b into a temporalstorage memory 8, and gradually transfers the picked-up image to theimage processing memory 9. An example in which the cameras 4 a and 4 bare attached to, for example, the vehicle to image an anterior area willbe described. However, the cameras 4 a and 4 b are not always attachedto the vehicle, but may be attached to a building.

As illustrated in FIG. 9, the image processing apparatus 1 includes twocameras of the cameras 4 a and 4 b, and further includes the memory 8,I/F 12 (image processing/transfer means), and a bus 10 that connects theimage input means 5, the memory 8, and the I/F 12, in addition to theconfiguration of the first embodiment. Also, in the second embodiment,the case in which the two cameras are used will be described, however,one, three or more cameras may be provided.

As in the first embodiment, as illustrated in FIG. 3, the imageprocessing apparatus 1 may be configured to be divided into the cameraunit 2 and the processing unit 3 such as a PC. For example, the camera 4a, the camera 4 b, the image input means 5, the memory 8, and the I/F 12may be configured to be installed in the camera unit 2, and the CPU 6,the memory 9, and the ROM 7 may be configured to be installed in theprocessing unit 3. In this case, the I/F 12 may be also provided in theprocessing unit 3, and execute a data communication with the camera unit2.

The image processing apparatus 1 can be used as device in which, forexample, as illustrated in FIG. 10, the cameras are attached thereto sothat viewing fields of the cameras overlap with each other, and adifference 8 (disparity) in hue when the image is picked up by therespective cameras is obtained to measure a distance. Symbol Z denotes adistance (distance to a point imaged to each of the pixels) to eachmeasurement point, f is a focal length, and b is a distance between thecameras.

In this embodiment, as illustrated in a functional block diagram of theimage processing apparatus 1 in FIG. 8, the camera 4 a, the camera 4 b,and an image transfer unit 24 are further provided as compared with thefirst embodiment of FIG. 1.

In the image processing apparatus 1, when a power supply turns on, theimage processing and a program for controlling the image processing areexecuted, and the image processing apparatus 1 functions as the imagecapturing unit 20, the image transfer unit 24, the processingsequence/frequency control unit 21, the processing region determinationunit 22, and the image processing unit 23, as illustrated in thefunctional block diagram of the image processing apparatus 1 in FIG. 8.

The image capturing unit 20 has a function of capturing the images fromthe two cameras (camera 4 a, camera 4 b), and storing the images in thememory 8 through the bus 10, unlike the operation of the image capturingunit in the first embodiment.

The image transfer unit 24 has a function of transferring the imagesthat have been captured in the memory 8 to the memory 9 through the bus11. In this situation, there may be applied a configuration in which therespective processing regions are set for the right and left imagespicked up by the camera 4 a and the camera 4 b in advance, and the imagetransfer unit reads those processing regions from the memory 9 tocalculate the disparity, and transfers at least any one of thecalculated disparity data and the image of the read processing region tothe memory 8. The results of implementing a variety of image processingsuch as edge detection processing, difference processing, optical flowcalculation processing, filter processing, and imageaddition/subtraction processing, or both of the implemented results andthe images within the processing region before the implementation aretransferred to the memory 8, in addition to the disparity calculationprocessing that is implemented by the image transfer unit 24.

The image transfer unit 24 has a function of transferring the respectiveprocessing regions determined by the processing region determinationunit 22 to the memory 9 through the bus 11 according to the sequence,the frequency, or the timing of the images captured in the memory 8 bythe image capturing unit 20 which is determined by the processingsequence/frequency control unit 21 through the bus 11.

The image processing apparatus 1 described above operates under thecontrol of an image processing control program, for example, asillustrated in FIG. 11. First, the overall or partial image picked up bythe camera is saved in the memory 8 through the bus 10 at the time t.Thereafter, an image in the first processing region in which the overallimage saved in the memory 8 at the time t+1 is reduced is transferred tothe memory 9. An image in the second processing region in which thevicinity of a center portion of the image saved in the memory 8 at atime t+2 is cut out is transferred to the memory 9. An image in which athird processing region of a left portion of the image saved in thememory 8 at a time t+3 is cut out is transferred to the memory 9. Then,an image in a fourth processing region which is located above the centerportion of an image saved in the memory 8 at a time t+4 is transferredto the memory 9.

As a result, as compared with a case in which images large in data sizeare captured in the memory 9 in a lump, the size of the image data to betransferred can be reduced, and the image in the transferred processingregion can be processed while a certain processing region is beingtransferred to the memory 9. For that reason, the processing time can bereduced.

A processing flow according to the second embodiment is illustrated inFIG. 12. Step 101 to Step 103, and Step 111 will be omitted from thedescription because those steps are common to those in the firstembodiment.

In Step 112, at least any one of the starting point coordinates, theending point coordinates, the reduction/enlargement ratio, the timing,the frequency, and the sequence of the processing regions is determinedwith the use of at least any one of the starting point coordinates, theending point coordinates, the capturing time, and the processing time ofthe respective processing regions illustrated in FIG. 7. For example,the viewing fields of the cameras are first most widely covered from thestarting point coordinates and the ending point coordinates of theprocessing region while the processing region in which a value of thereduction/enlargement ratio reduces the image to ½ is first transferred,on the basis of the respective processing regions illustrated in FIG. 7.Then, in Step 113, the image is captured according to the starting pointcoordinates, the ending point coordinates, the reduction/enlargementratio, the timing, the frequency, and the sequence of the respectiveprocessing regions set in Step 112.

For example, it is assumed that the first processing region is firsttransferred. Then, the processing region in which the transferprocessing is completed during the processing time of the firstprocessing region is selected (it is assumed that the second processingregion is selected), and second transferred. Also, likewise, theprocessing region in which the transfer processing is completed duringthe processing time of the second processing region is selected (it isassumed that the third processing region is selected), and thirdtransferred. Further, likewise, the processing region in which thetransfer processing is completed during the processing time of the thirdprocessing region is selected (it is assumed that the fourth processingregion is selected), and fourth transferred. In this case, for example,when it is assumed that the fourth processing region includes the thirdprocessing region, a portion of the fourth processing region whichoverlaps with the third processing region is not transferred, but onlythe remaining fourth processing region may be transferred.Alternatively, when for example, the fourth processing region isincluded in the third processing region, or in the second and thirdprocessing regions (included in one or more processing regions), thefourth processing region is not transferred, but the second processingregion and the third processing region are transferred in advance, andthe transfer of the fourth processing region can be omitted.

Then, the first processing region is transferred from the memory 8 tothe memory 9 in Step 114, and the image in the first processing regionstored in the memory 9 is processed to conduct lane detection processingin Step 115. Further, in Step 116, the image in the first processingregion is processed to detect candidates of the leading vehicle and thesigns. In this case, the second processing region and the thirdprocessing region may be configured to be determined on the basis of thecoordinates and the sizes of the candidates of the leading vehicle andthe signs.

Thereafter, the second processing region is transferred in Step 117, andthe leading vehicle is detected with the use of the image in the secondprocessing region in Step 118. In Step 118, if the position of theleading vehicle is roughly specified on the basis of the result of Step116 to limit the region to be processed, the more efficient processingis enabled.

The third processing region is transferred in Step 119, and the signsare detected in Step 120. Then, the fourth processing region istransferred in Step 121, and the detection processing of a traffic lightis executed in Step 122. In this situation, the signal light detectionprocessing is executed with the use of both of the third processingregion and the fourth processing region which are stored in the memory 9so that the traffic light can be more extensively detected. Finally, thestarting point coordinates, the ending point coordinates, the frequency,the timing, and the sequence of the first to fourth processing regions,which are processed in the subsequent picked-up image, are updated onthe basis of the detection results of the lane, the leading vehicle, thesigns, and the traffic light (Step 123).

As described above, in the image processing apparatus and the controlmethod of the image processing apparatus according to the secondembodiment, for example, when the imaging element of the high resolutionis used, because all of the pixels in the imaging element is notcaptured in the memory 9 that stores the image to be processed by theCPU, the transfer load of the bus 11, and the processing load of the CPU6 can be reduced. In this situation, the processing region for reducingthe region wide in the image as in the first processing region, and theprocessing regions in which the image is cut out without reducing theimage as in the second to fourth processing regions are set so thatsensing wide in the viewing angle and sensing high in the resolution canbe efficiently realized at the same time. For example, as illustrated inFIG. 13, the processing region in which the region wide in the image isreduced is captured at the time t+1, the reduced image is processed atthe time t+2, the lane is detected, and candidate regions in which thevehicle, the sign, and the signal light are likely to exist aredetected. In parallel to this processing, the processing region in whichthe center portion of the image is not reduced is captured, thedetection processing of the vehicle is conducted on the basis of thecandidate region in which the vehicle is likely to exist at the timet+3. In parallel to this processing, the processing region including thecandidate region of the signs is captured without being reduced. Then,the detection processing of the signs is conducted at the time t+4, andthe processing region located above the center of the image is capturedwithout being reduced. Thereafter, the captured image is processed todetect the signal light.

Also, as illustrated in FIG. 14( a), in the processing region used forprocessing of the lamp detection such as ahead light or a tail lamp,monochrome and color are set at the same reduction/enlargement ratio. Asillustrated in FIG. 14( b), in the sign detection processing, themonochrome and the color components may be set at differentreduction/enlargement ratios such that the monochrome component is highin resolution and the color component is low in resolution. This isbecause, for example, in the sign detection, the resolution of colorinformation may be low although the color information is important inthe processing of finding out the signs, and numbers need to be clearlyimaged more than the color information in the processing of identifyingthe sings (identification of speed limit 40 km and speed limit 60 km).Also, as illustrated in FIGS. 14( c) and (d), in the vehicle detectionand the pedestrian detection, in order to capture a change in thebrightness of the vehicle or the pedestrian in detail, the monochrome iscaptured with high resolution, and the color is captured withinformation low in resolution. Also, as illustrated in FIG. 14( e), inthe vehicle detection, only monochrome is captured. In this case, whenthe color of the lane is determined, color information may be captured.In the signal light detection as illustrated in FIG. 14( f), when adistant signal light is detected, both of monochrome and color arecaptured with the high resolution. Finally, as illustrated in FIG. 14(g), the image for processing the overall image is captured as the lowresolution in both of the monochrome and the color. The resolution ofthe color and monochrome is changed according to an object to berecognized to enable more efficient processing.

Also, the mobile object to which the cameras are attached may be anymovable object such as robots, construction machines, or farm machinesin addition to the vehicle. The processing of the image processing unit23 may include processing for correcting a dark image to a bright imagefor showing the person the image, processing of converting the startingpoint of the camera, processing of reducing the noise of the image,processing of improving the contrast of the image, processing ofcorrecting blur of the image, or processing for converting the image toan edge image. Also, in the processing of detecting an object from theimage by the image processing unit 23, the object includes animalsincluding the persons, mobile bodies such as vehicles, three-dimensionalobjects such as walls or power poles, regions that changes in the image,abnormal actions (crimes, etc.) of the person, regions different inmotion such as only one person travel to the left when a lot of personstravel to the right, traffic signs painted on roads such as the lanes orcrosswalks, head lights, tail lamps, and lamps such as traffic lights.

In this way, the processing regions are set according to the type ofimage processing (the lamp detection processing, the sign detectionprocessing, the vehicle detection processing, the pedestrian detectionprocessing, the lane detection processing, and the signal lightdetection processing) executed by the image processing unit. Asillustrated in FIG. 14, the processing regions are set at the differentpositions (coordinates) according to the type of the image processing.

As illustrated in FIG. 15, the processing sequence/frequency controlunit 21 first captures the processing region long in the processingtime, and subsequently captures the processing region long in a totaltime of the capturing and transfer time. As a result, the processingsequence/frequency control unit 21 captures and transfers the processingregion long in the capturing and transfer time while processing theprocessing region long in the processing time to enable the efficiencyof the processing. Also, when the image transfer unit 24 is not providedas in the first embodiment, the sequence, the timing, or the frequencycan be determined according to the capturing time and the processingtime of the processing region. The processing region takes a time tocapture when the data size of the image in the processing region islarge, or when it takes time to calculate the processing for allowingthe image capturing unit to correct the brightness in capturing theimage. Likewise, the processing region takes a time to transfer when thedata size of the image in the processing region to be transferred islarge, or when it takes a time to calculate the image processing whenthe image is processed and then transferred by the image transfer unit24

For example, as illustrated in FIG. 15, the first processing region iscaptured at the time t+1, transferred at the time t+2, and processed atthe time t+3. This processing is, for example, a process of processingthe overall image, and detecting the vehicle, the pedestrian, the lane,the signs, the signal light, and the candidates thereof. Then, at thetime t+2, and the time t+3, the second processing region is captured andtransferred. Then, the transferred image is processed at the time t+4.This processing is, for example, a process of capturing the image withhigh resolution, and detecting the pedestrian or the distant vehicle.Then, after the capturing of the second processing region has beencompleted, the third processing region is captured, transferred, andprocessed. This processing is a process of the sign detection or thesignal light detection. In this way, the sequence, the timing, etc. arecontrolled taking the capturing time, the transfer time, and theprocessing time into account with the result that a time during whichthe CPU does not operate can be shortened.

The invention claimed is:
 1. An image processing apparatus comprising:an image capturing unit that captures image information picked up by animaging element; a processing region setting unit that sets a pluralityof processing regions for the image information; a processingsequence/frequency determination unit that determines at least asequence, a frequency, and a timing of capturing the respective imageinformation in the plurality of set processing regions, and at least asequence, a frequency, and a timing of processing the respective imageinformation; and an image processing unit that captures the imageinformation for each of the processing regions according to thesequence, the frequency, and the timing determined by the processingsequence/frequency determination unit, and processes the captured imageinformation according to the sequence, the frequency, and the timingdetermined by the processing sequence/frequency determination unitwherein the sequence, the frequency, and the timing determined by theprocessing sequence/frequency determination unit are determined on thebasis of an exposure time, starting point coordinates, ending pointcoordinates, a capturing time, a processing time, and areduction/enlargement ratio of the respective processing regions set bythe processing region setting unit.
 2. The image processing apparatusaccording to claim 1, wherein the image capturing unit has a function ofsetting a shutter or a gain which is exposure information of the imagingelement.
 3. The image processing apparatus according to claim 1, whereinthe image capturing unit has a function of executing at least oneprocessing of edge emphasis processing, noise removal processing, andbrightness correction processing on the captured image information. 4.The image processing apparatus according to claim 1, wherein theprocessing region setting unit sets the processing regions according toa type of image processing executed by the image processing unit.
 5. Theimage processing apparatus according to claim 4, wherein the processingregion setting unit sets the processing regions at different positionsaccording to a type of image processing executed by the image processingunit.
 6. The image processing apparatus according to claim 1, whereinthe image capturing unit captures the image information of a secondprocessing region different from a first processing region according tothe sequence, the frequency, or the timing determined by the processingsequence/frequency determination unit while the image processing unit isprocessing the image information of the first processing regionaccording to the sequence, the frequency, or the timing determined bythe processing sequence/frequency determination unit.
 7. The imageprocessing apparatus according to claim 1, further comprising: an imagetransfer unit that transfers the image information captured by the imagecapturing unit to the image processing unit according to the sequence,the frequency, or the timing determined by the processingsequence/frequency determination unit.
 8. The image processing apparatusaccording to claim 1, wherein the image capturing unit captures aplurality of image information of the processing regions different inexposure during one exposure period.
 9. An image processing apparatuscomprising: an image capturing unit that captures image informationpicked up by an imaging element; a processing region setting unit thatsets a plurality of processing regions for the image information; aprocessing sequence/frequency determination unit that determines atleast any one of a sequence, a frequency, and a timing of capturing therespective image information in the plurality of set processing regions,and at least any one of a sequence, a frequency, and a timing ofprocessing the respective image information; and an image processingunit that captures the image information for each of the processingregions according to the sequence, the frequency, or the timingdetermined by the processing sequence/frequency determination unit, andprocesses the captured image information according to the sequence, thefrequency, or the timing determined by the processing sequence/frequencydetermination unit, wherein the processing region setting unit sets theprocessing regions at different positions according to whether the imageprocessing executed by the image processing unit is sign detectionprocessing or pedestrian detection processing.
 10. An image processingapparatus comprising: an image capturing unit that captures imageinformation picked up by an imaging element; a processing region settingunit that sets a plurality of processing regions for the imageinformation; a processing sequence/frequency determination unit thatdetermines at least any one of a sequence, a frequency, and a timing ofcapturing the respective image information in the plurality of setprocessing regions, and at least any one of a sequence, a frequency, anda timing of processing the respective image information; an imageprocessing unit that captures the image information for each of theprocessing regions according to the sequence, the frequency, or thetiming determined by the processing sequence/frequency determinationunit, and processes the captured image information according to thesequence, the frequency, or the timing determined by the processingsequence/frequency determination unit; and an image transfer unit thattransfers the image information captured by the image capturing unit tothe image processing unit according to the sequence, the frequency, orthe timing determined by the processing sequence/frequency determinationunit, wherein the image transfer unit transfers a processing regionlonger in transfer time among the plurality of processing regions to theimage processing unit while the image processing unit is processing theprocessing region longer in processing time.